



MAT. 






1 Ettfc 


DOSSIER 

paiTChes PaLuiiLlllll ' 





(19) 



European Patent Office 
Office europeen des brevets 



(11) 



EP 1 102 241 A1 



(12) 



EUROPEAN PATENT APPLICATION 



(43) 


Date of publication: 


(51) lntCl7: G10L 15/22 




23.05.2001 Bulletin 2001/21 






Annliofltinn number 6 




(22) 


Date of filing: 19.11.1999 




(84) 


Designated Contracting States: 


(72) Inventor: Janssen, DIonyslus Paul Marie 




AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


5081 BC Hitvarenbeek (NL) 




MC NL PT SE 






Designated Extension States: 


(74) Representative: Griebling, Onno et al 




AL LT LV MK RO SI 


Exter Polak & Charlouis B.V., 






P.O. Box 3241 


(71) 


Applicant: Medical Development & Technology 


2280 GE Rijswijk (NL) 




Information Division B.V. 






5062 CD Oisterwijk (NL) 





(54) Adaptive voice-controlled dialogue system 

(57) An adaptive speech-controlled dialogue sys- 
tem (1 ) is described, which is arranged for adapting a 
dialogue characteristic to specific wishes of a specific 
user based on experiences with respect to this specific 
user. The system can be in a predetermined number of 
dialogue states (DSi), wherein each di alogue state (DSi) 
is associated with predetermined dialogue actions, in- 
cluding predetemnined transition possibilities (TR(i;j)) to 



predetermined other dialogue states (DSj). In at least 
one dialogue state, the system has certain communica- 
tion options with respect to the communication with the 
user, wherein the system adapts itself in an adaptive 
way to apparent wishes of the user concerned. 

Particularly, the system is an^anged for developing, 
for at least one user (U), a user profile which defines for 
the said communication options the preferences asso- 
ciated with that said user (U). 
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Description 

[0001] The present invention relates in general to a 
speech-controlled dialogue system, with which is meant 
a computer system that can exchange data with a user 5 
and can receive instructions and data in dialogue form, 
especially by means of spoken text. 
[0002] Computer systems for recording data and later 
retrieval of data are commonly known. Such a system 
has in general a communication interface for two-way io 
communication with a user. For communication from us- 
er to system, hereinafter indicated as "input", such com- 
munication interface comprises input means, and for 
communication from system to user, hereinafter indteat- 
ed as "output", such communication interface compris- ^5 
es output means. Conventionally, the output means are 
primarily a display and a printer, while the input means 
are, conventionally, primarily a keyboard. 
[0003] For increasing the ease of use, speech-based 
communication interfaces have recently been devel- 20 
oped. Herein, the input means comprise a microphone 
for receiving spoken text from a user, as well as a 
speech recognition module, which converts the sound 
signals received from the user into signals understand- 
able for the system, such as for instance text or control 25 
instructions. Conversely, the output means can com- 
prise a speaker or the like for generating sound signals, 
as well as a speech generation module generating 
speech signals. Then, for instance, a user may, instead 
of reading a written message "close file (yes/no)?" on 30 
his screen, hear a spoken text "do you want to close the 
file?", and he may, instead of typing the instruction "yes 
[enter]", pronounce the instruction "yes". Since speech- 
based communication interlaces are known per se, for 
instance from US-A-5.051 .924 or US- A-5.1 68.548. they 35 
will not be explained further here. 
[0004] Although a speech-controlled dialogue system 
may already be used in the case of only one single user, 
and the present invention relates also to such a situa- 
tion, the present invention more particularly relates to a 40 
speech-controlled dialogue system with a large number 
of users that can input and/or request infomnation 
through that system. An important example of an appli- 
cation situation is a hospital, where it must be possible 
for medical files to be looked into and/or to be modified ^5 
at different places by different persons. The present in- 
vention relates particulariy, but not exclusively, to such 
an application situation, and will therefore be explained 
hereinafter for this application example. However, it is 
stressed that the invention can also be applied to other so 
areas, such as for instance an insurance office, an ad- 
ministration office, etc. 

[0005] Existing systems have the disadvantage that 
they follow a dialogue with a user according to a prede- 
termined and fixed protocol. The protocol, therefore, is 55 
not easy or convenient in use for all users to the same 
extent. In any case, each user must get used to a non- 
changing system. 



[0006] The present invention aims to solve this prob- 
lem. 

[0007] More particulariy, the present invention aims to 
provide a dialogue system which provides increased 
convenience of use in that the dialogue followed by the 
system is better tuned to the wishes of each individual 
user. 

[0008] In principle, it would of course be possible to 
consider the dialogue system as a combination of many 
dialogue subsystems, wherein each subsystem is allo- 
cated to a fixed user and follows a preprogrammed di- 
alogue characteristic which is adapted to this one fixed 
user. Such subsystems could be implemented in hard- 
ware or software. A disadvantage of such an approach 
is, however, that a new dialogue characteristic must be 
developed for each new user. 

[0009] The present invention chooses another ap- 
proach. According to an important aspect of the present 
invention, a dialogue system is adaptive. The dialogue 
characteristic is not fixed but is adapted to the specific 
wishes of the users, by the system itself, based on ex- 
perience relating to each user. 

[0010] In practice, this has as result that each user 
may experience the system in an individual way as be- 
ing a dialogue system fitting him well, and adapted to 
his specific wishes. Instead of a user having to get used 
to a fixed system, he experiences a flexible system 
which gets used to the user. Based on experiences from 
the past, the system can even anticipate to wishes of 
the user. 

[0011] These and other aspects, characteristics and 
advantages of the present invention will be explained 
further by the following description of an application ex- 
ample of a dialogue system according to the invention 
with reference to the drawing, in which: 

figure 1 shows a block diagram of a dialogue sys- 
tem; and 

figure 2 illustrates dialogue states and transitions 
there between. 

[0012] Figure 1 schematically shows a dialogue sys- 
tem indicated generally with the reference number 1 , ap- 
plied in the context of a hospital 2 for processing medical 
files. A large number of users U, only three of which are 
shown in figure 1 , may be connected to the system 1 . A 
usercan be connected directlyto the system 1 by means 
of a temriinal 3 arranged within the hospital 2, such as 
shown for a first user U1 . Such a user will be indicated 
as "intemal" user. However, the system 1 is also acces- 
sible for external users. In figure 1, a second user U2 
(for instance a physician) is shown, which is connected 
to the system by means of a telephone network 4, and 
a third user U3 is shown, which is coupled to the system 
by means of the internet 5. However, it will be clear that 
a dialogue system may offer one or more of said cou- 
pling possibilities simultaneously. 
[0013] The dialogue system 1 proposed by the 
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present invention can be in a predetemnlned number of 
dialogue states, which will be indicated hereinafter as 
DSi, wherein i is an integer larger than zero. The number 
of possible dialogue states DSi is large, but finite. Each 
dialogue state DSi is characterized by one or more of 
the following dialogue actions to be perfomned by the 
system: 

a) Presenting Information to the user. 

b) Posing a question to the user 

c) Receiving input from the user. 

d) Taking an action in response to the input (c). 

e) Going to a next dialogue state in response to the 
input (c). 

[0014] The dialogue actions (a) and (b) concern an 
infomriation transfer from the system to the user. In ac- 
tion (a), the infomnatlon may for instance be presented 
on a display or a screen, but the information may also 
be presented auditively in the fomn of speech. In action 
(b), a question can be posed in written fomn, for instance 
by projection on a screen, but preferably a question is 
posed auditively in the form of speech. 
[0015] The dialogue action (c) concerns an infomria- 
tion transfer from the user to the system. The input can 
concern data to be processed, but may also be a com- 
mand or a question, respectively; preferably, input takes 
place through spoken text. 

[0016] The dialogue action (d) concerns an action not 
necessarily involving infonnation transfer from and/or 
towards the user. 

[0017] A transition from a dialogue state DSi to a next 
dialogue state DSj will be indicated as transition TR(i;j). 
[0018] Figure 2 illustrates schematically a number of 
dialogue states DS1 to DS6 in the form of circles, and 
possible transitions in the form of arrows. Next to the 
arrow which connects dialogue state DS1 with dialogue 
state DS4, the indication TR(1 ;4) is shown; the other 
transition indications are left out for the sake of simplic- 
ity. It appears clearly from figure 2 that the system 1 can 
reach a certain dialogue state (for instance 084) from 
different previous dialogue states (for instance DS1, 
□82). Further it appears clearly from figure 2 that, from 
a certain dialogue state (for instance DS1), the system 
has in principle a plurality of transitions available (for In- 
stance TR(1;4), TR(1;5), TR(1 ;6)) for reaching a next 
dialogue state (DS4, DS5, DS6, respectively); the sys- 
tem will decide, based on the input received from the 
user, which transition is actually made. 
[0019] In many dialogue states, the system has cer- 
tain communication options regarding the communica- 
tion with the user. Regarding the infonnation transfer 
from the system to the user (the dialogue actions (a) and 
(b)), the system has for instance the following options: 

which information to present or which question to 
pose, respectively; 

which order to keep when presenting information or 



posing questions, respectively. 

[0020] Regarding the information transfer from the us- 
er to the system (the dialogue action (c)), the system 
5 has for instance the choice of how a particular request 
must be interpreted. 

[0021] Further, in many dialogue states, the system 
has certain action-options regarding an action to be tak- 
en (dialogue action (d)) or a transition to be executed 

10 (dialogue action (e)). 

[0022] According to an important aspect of the 
present invention, the dialogue system 1 is, at least in 
at least one dialogue state, not fixed in executing the 
above-mentioned options but the system confonns itself 

IS in an adaptive way to the apparent wishes of the user 
concerned. More particulariy, the system remembers 
which choices the user has made in the past in the same 
circunnstances, and the system considers that the user 
now wants to make the same choice. Thus, the system 

20 develops for each user a user profile which defines for 
the above-mentioned option preferences associated 
with a certain user. During each dialogue session be- 
tween system and user, first the identity of the user is 
established by the system, the user profile associated 

25 with that user is activated, and that activated user profile 
is used for making choices. Further, if necessary, the 
user profile will be amended. 

[0023] For recording and remembering the user pro- 
file, the system comprises a memory which is not shown 

30 for the sake of simplicity. 

[0024] Hereinafter, the present invention will be fur- 
ther explained in the light of some examples, wherein it 
will be assumed that the communication between user 
and system takes place through speech, and wherein 

35 always will be assumed that no speech confusion aris- 
es. Further, the dialogue system will be indicated by the 
character 8 and the user will be indicated with the char- 
acter U. 

[0025] In a first dialogue state, the identity of the user 
40 is examined. For instance, that can be done as follows: 

U: "new user". 

S: "what is your name?" 

U: "Janssen". 

45 

[0026] Then, in a second dialogue state, the system 
can execute a verification step, wherein some personal 
data of the user are mentioned, and wherein the user is 
asked for confirmation, and finally the system can ask 

50 a password or verification code. For reasons of security 
and for protecting the privacy of the user, these data are 
preferably not exchanged by means of speech but in a 
way which is not easily overheard by those accidentally 
present, for instance through screen and keyboard. 

55 [0027] When the system has established the identity 
of the user, the system enters a third dialogue state, 
wherein the user is asked what he now wants to do, for 
instance as follows: 
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S: "what do you want to do?". 

[0028] There is of course, a large amount of possibil- 
ities for the user, but for the sake of simplicity it will be 
assumed that the user can chose from the following ac- 
tions: 

1) making an examination report; 

2) examining lab results; 

3) writing a report letter; 

4) writing a referral letter; 

5) requesting an examination. 

[0029] Suppose that the user is a family doctor who 
wishes to make a referral letter, and who answers as 
follows: 

U: "letter. 

[0030] Here, the system finds insufficient infomriation 
in the input for making a choice: the possibilities (3) and 

(4) are the most probable, but the possibilities (1) and 

(5) cannot be excluded. Then, the system is forced to 
obtain further information. 

[0031] Conventionally, a computer program returns to 
a question step if the answer to the question does not 
meet the criteria set by the program. In this case, this 
would mean that the system would repeat its question, 
and that the above-described cycle of question and an- 
swer can stay repeating itself. 

[0032] In order to make clear to the user what is ex- 
pected, the dialogue system might follow the options 
one by one in a fixed order, and ask if that is the right 
choice, as follows: 

S: "do you want to make an examination report, yes 

or no?" 
U: "no" 

S: "do you want to examine lab results, yes or no?" 
U: "no" 

S: "do you want to write a report letter, yes or no?" 
U: "no" 

S: "do you want to write a refen^al letter, yes or no?" 
U: "yes" 

[0033] Of course, this cycle of questions does lead to 
the result that the wish of the user is made clear, such 
that the system can go on to a next dialogue state, but 
it is inconvenient to the user that only the fourth question 
is a hit. In practice, of course, the number of possibilities 
is larger, and so it can take longer before the system 
reaches the possibility wished by the user. 
[0034] According to an important aspect of the 
present invention, the system is adapted for remember- 
ing the answer of the user, and for incorporating the an- 
swer in the user profile belonging to this user U. When 
this same user U reports a next time, the system will 
pose its question, based on earlier experiences record- 



ed in the user profile. After all, apparently this user U 
meant "referral letter" with the command "letter and the 
chances are that is what he means now again. A next 
time, the system will pose this question first. 

5 [0035] The system may be adapted for adapting each 
choice to the previous meaning of the command con- 
cerned, but preferably the system is adapted to adapt 
its choice to the largest number of previous meanings 
of the command concerned. If, after six "referral letters", 

10 the user writes a report letter ones, a next time the sys- 
tem will still assume that the command "letter means 
"refenral letter. 

[0036] After a learning phase, in which the system 
"gets used" to the user, the system will "recognize" this 

15 command "letter better and better in a way adapted to 
the user concerned. With a different user, the recogni- 
tion can progress differently. Suppose that a second us- 
er exists, a surgeon who, with "letter, always means 
"report letter; in time, the system will associate the com- 

20 mand "letter with the meaning "report letter with this 
second user, and will incorporate that in the user profile 
belonging to this second user. 

[0037] In this way. all users are offered the important 
convenience of use that the dialogue system "under- 

25 stands" what the user means by certain instructions. 
[0038] In the end it may even be so that the dialogue 
system recognizes with the first-mentioned user (gen- 
eral practitioner) that he will almost always request a 
referral letter, and only seldom ly something else. Then, 

30 the dialogue system can adapt its first question in this 
dialogue state to this, for instance by, instead of "what 
do you want to do?", posing as first question: 

S: "do you want to make a referral letter again?" 

35 

[0039] Another example concems a specialist who 
wants to refer a patient to a radiologist for making an X- 
ray photo. Then, the system enters a dialogue state in 
which further information is asked about the X-ray photo 
40 to be made; in the referral letter must, inter alia, be in- 
cluded from which part of the body the photo must be 
made. The system can ask the specialist for Infomnation, 
as follows: 

45 S: "which part of the body?" 
U: "lungs" 

[0040] it will however be clear that a lung specialist is 
generally more interested in lung photos than in photos 

50 from other body parts such as knees. Because the lung 
specialist primarily chooses for lung photos, the system 
according to the present invention incorporates this 
preference in the user profile of this user/lung specialist, 
and it will adapt the dialogue thereto. Then, for instance, 

55 the dialogue can be: 

S: "lung photo?" 
U: "yes" 
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[0041] An important advantage according to the 
present invention is the increased ease of use in that 
exchange of infonnation takes place by means of 
speech, at least in as far as it concerns the direct transfer 
from and towards a user. In the system itself, and/or in 
a communication network, the transfer can take place 
by means of other, more suitable coding. For instance, 
in the case of communication through internet, the 
speech signals can be converted into HTML codes. 
[0042] Generating speech by a system for communi- 
cation with the user can be perfomned in several ways. 
Byway of example it is possible that a sequence of char- 
acters (letters, numerals), for instance ASCII-charac- 
ters, generated by the system, are converted into 
speech signals by a speech converter at the place of the 
user. However, it is also possible that use is made of 
preprogrammed speech sequences, which can be acti- 
vated by the system as desired by sending a regarding 
activation code ("canned speech"). An advantage asso- 
ciated with this latter method is that the system can con- 
fine itself to generating a relatively short code (for in- 
stance. code-1), while at the place of the user a rela- 
tively long sentence, containing much Infomnation, is 
generated (for instance: "what do you want to do?"). 
[0043] A further important advantage of the system 
according to the present invention is the flexible inter- 
action with the user. In the foregoing it has already been 
explained that the system bases itself, when making 
choices, on the user profile associated with the user 
Further, the system is very flexible in accepting input by 
the user. More particularly, the user is not obligated to 
fomnulate his commands and data input according to a 
predetermined format. In the foregoing, it has been de- 
scribed by way of example that a user reports himself 
with the message "new user". However, the user is not 
bound to report himself in this way. The system is ar- 
ranged for, within certain limits, understanding the se- 
mantics of the user's command, for instance by recog- 
nition of certain key words (for instance: "1 am a new 
user"), or recognition of equivalence (for instance: "I 
wantto begin"). Such keywords and/or equivalence can 
be preprogrammed. However, also in this respect it is 
possible that the system adapts itself to the user by in- 
corporating earlier non-understood input in a table of 
equivalence in the user profile after elucidation. A user 
who reports himself with for instance the text "hello" , will 
maybe not be understood directly. By means of a cycle 
of question/answer, the system can then learn the inten- 
tion of the user. By regarding the association between 
this intention and the text "hello", a next time this user 
reports himself with the text "hello" the system will au- 
tomatically be able to interpret this as being a possible 
equivalent for "new user". 

[0044] Since it will be clear to persons skilled in the 
art how the system according to the present invention 
can be implemented by suitable software and/or hard- 
ware, this aspect will not be explained here further 
[0045] It will be clear to a person skilled in the art that 



the scope of the present invention Is not limited to the 
examples discussed in the foregoing, but that several 
amendments and modifications thereof are possible 
without deviating from the scope of the invention as de- 
5 fined In the attached claims. 



Clainns 

10 1. Adaptive speech-controlled dialogue system (1), 
comprising a speech -based communication inter- 
face through which a user (Ul; U2; U3) can receive 
and input data and can input instructions, wherein 
the system is arranged for adapting a dialogue char- 
ts acteristic to specific wishes of a specific user based 
on experience in relation to this specific user. 

2. Dialogue system according to claim 1 , wherein the 
communication interface uses a network such as a 

20 telephone network (4) or the internet (5). 

3. Dialogue system according to claim 1 or 2, wherein 
the system can be in a predetermined number of 
dialogue states (DSi), wherein each dialogue state 

25 (DSi) is associated with predetemnined dialogue ac- 
tions, including predetenriined transition possibili- 
ties (TR(i;j)) to predetermined other dialogue states 
(DSj). 

30 4. Dialogue system according to claim 3, wherein, In 
at least one dialogue state, the system has certain 
communication options with respect to the commu- 
nication with the user 

35 5, Dialogue system according to claim 4, wherein, at 
least in respect of at least one of said dialogue 
states where the system has communication op- 
tions, the system adapts itself in an adaptive way to 
apparent wishes of the user concerned. 

40 

6. Dialogue system according to claim 5, wherein the 
system is arranged for, for at least one user (U), de- 
veloping a user profile which defines for the said 
communication options the preferences associated 

45 with that said user (U). 

7. Dialogue system according to claim 6, wherein the 
system is arranged for, at least in respect of the said 
one dialogue state, incorporating an answer or 

50 choice given by the user into the user profile belong- 
ing to this user (U). 

8. Dialogue system according to claim 6 or 7, wherein 
the system is arranged for posing questions in a cer- 

55 tain situation based on the earlier experiences with 
respect to the same situation embedded in the user 
profile. 
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9. Dialogue system according to claim 6 or 7, wherein 
the system is anranged for interpreting an instruc- 
tion of the user (U) in a certain situation based on • 
the earlier experiences with respect to the same sit- 
uation incorporated in the user profile. ^ 

10. Dialogue system according to claim 8 or 9, wherein 
the system is arranged for incorporating Into the us- 
er profile exclusively the last previous experience 
with respect to a certain situation. 

1 1 . Dialogue system according to claim 8 or 9, wherein 
the system is arranged for embedding into the user 
profile a predetemnined number of previous experi- 
ences with respect to a certain situation, and to 15 
base its choice on the largest number of experienc- 
es incorporated in the user profile. 
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